Scorecard Integration #1294

404-geek · 2024-06-26T11:05:43Z

ScoreCode Integration

This pull request integrates the ScoreCode Repo into SCIO, enabling the fetching of the latest OSSF Scorecard Data for discovered packages using their vcs_url. The current implementation supports github.com and gitlab.com VCS URLs.

Key Features:

Integration with ScoreCode Repo
Fetching of OSSF Scorecard Data using vcs_url
Support for github.com and gitlab.com VCS URLs

Related Issues:

This feature enhances SCIO's functionality by ensuring that users can retrieve the most up-to-date security scores for packages discovered in their projects, improving overall security assessment and management.

developed functions to check for availability nexB#598 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…aving logic nexB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…exB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

… nexB#598 Signed-off-by: 404-geek <[email protected]>

…ecard_integration

…up.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…gration # Conflicts: # scanpipe/models.py # scanpipe/tests/test_models.py

Signed-off-by: 404-geek <[email protected]>

scanpipe/migrations/0068_merge_20240820_1656.py

scanpipe/models.py

tdruez · 2024-08-21T05:12:15Z

scanpipe/models.py

+    )
+
+    @classmethod
+    @transaction.atomic()


@404-geek Could you provide an example that shows why atomic() is useful here?

scanpipe/models.py

scanpipe/pipelines/get_scorecard_info_packages.py

tdruez · 2024-08-21T05:20:43Z

scanpipe/pipelines/get_scorecard_info_packages.py

+        if not scorecard.is_available():
+            raise Exception("scorecode service is not available.")
+
+    def lookup_save_packages_scorecode_info(self):


fetch_packages_scorecode_info would be better.

scanpipe/pipelines/get_scorecard_info_packages.py

setup.cfg

…#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

pombredanne · 2024-09-29T22:21:01Z

@404-geek can you check the failing tests?

tdruez · 2024-10-29T06:30:51Z

Hey @404-geek , what's your latest status on this PR? Any chances we can complete and merge it before it diverges too much from the main branch?

404-geek · 2024-10-29T17:36:52Z

Hey @404-geek , what's your latest status on this PR? Any chances we can complete and merge it before it diverges too much from the main branch?

Hi @tdruez,

I’m planning to have this PR ready by next week. Most of the requested changes have been addressed; I just need to add a few test cases before pushing the final version.

Signed-off-by: 404-geek <[email protected]>

…ecard_integration

Signed-off-by: 404-geek <[email protected]>

tdruez

@404-geek See my various comments ;)

Also, the new pipeline needs to be added to the built-in-pipelines.rst documentation.

tdruez · 2024-12-03T09:21:07Z

scanpipe/migrations/0070_alter_project_purl_discoveredpackagescore_and_more.py

+        migrations.AlterField(
+            model_name='project',
+            name='purl',
+            field=models.CharField(blank=True, help_text="Package URL (PURL) for the project, required for pushing the project's scan result to FederatedCode. For example, if the input is an input URL like https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz, the corresponding PURL would be pkg:npm/[email protected].", max_length=2048),
+        ),


Why is this field impacted by this PR?

tdruez · 2024-12-03T09:24:07Z

scanpipe/models.py

+    def __str__(self):
+        return self.score or str(self.uuid)
+


Follow the conventions used across the existing Models:

Fields

class Meta

str

tdruez · 2024-12-03T09:24:26Z

scanpipe/models.py

+    )
+
+    @classmethod
+    @transaction.atomic()


@404-geek You haven't address the question above yet ;)

tdruez · 2024-12-03T09:24:46Z

scanpipe/models.py

+
+
+class ScorecardCheck(UUIDPKModel, ScorecardChecksMixin):
+    def __str__(self):


Follow the conventions used across the existing Models:

Fields

class Meta

str

tdruez · 2024-12-03T09:25:29Z

scanpipe/models.py

+        blank=True,
+        null=True,


Can a DiscoveredPackageScore instance really exists without a DiscoveredPackage FK defined?

tdruez · 2024-12-03T09:30:59Z

scanpipe/pipelines/fetch_scorecode_info.py

+    def check_scorecode_service_availability(self):
+        """Check if the scorecode service is configured and available."""
+        if not ossf_scorecard.is_available():
+            raise Exception("scorecode service is not available.")


Use ScoreCode case for consistency.

tdruez · 2024-12-03T09:32:10Z

scanpipe/tests/__init__.py

@@ -284,3 +286,10 @@ def make_dependency(project, **extra):
        "license_key": "mpl-2.0",
    },
 }
+
+scorecard_data = None


That line seems unnecessary.

tdruez · 2024-12-03T09:33:28Z

scanpipe/tests/__init__.py

+data = Path(__file__).parent / "data"
+
+with open(f"{data}/scorecode/scorecard_response.json") as file:
+    scorecard_data = json.load(file)


This should not be loaded in the module init but as needed in the test function context.

tdruez · 2024-12-03T09:34:27Z

scanpipe/tests/test_models.py

@@ -2477,6 +2481,29 @@ def test_scanpipe_codebase_resource_queryset_elfs(self):
        self.assertTrue("e" in paths)
        self.assertTrue("a" in paths)

+    def test_scorecard_models(self):


The name of this function does not follow existing conventions.

tdruez · 2024-12-03T09:35:17Z

scanpipe/tests/test_pipelines.py

+
+        run = project1.add_pipeline(pipeline_name)
+        pipeline = run.make_pipeline_instance()
+        # mock_is_configured.return_value = True


Unwanted comment.

404-geek added 30 commits June 26, 2024 13:20

Add ScoreCard config into settings.py

173db43

developed functions to check for availability nexB#598 Signed-off-by: 404-geek <[email protected]>

code style fix nexB#598

272b99c

Signed-off-by: 404-geek <[email protected]>

Merge branch 'refs/heads/ossf_api_template' into scorecard_integration

944cee2

settings.py code style fix nexB#598

bc445c1

Signed-off-by: 404-geek <[email protected]>

mixin import and models declaration nexB#1283

f241b3b

Signed-off-by: 404-geek <[email protected]>

transforming scorecard data into object for saving nexB#1283

605a5cf

Signed-off-by: 404-geek <[email protected]>

added test cases for saving scorecard data into models and modified s…

3833dca

…aving logic nexB#1283 Signed-off-by: 404-geek <[email protected]>

Merge branch 'main' into feat-models_integration

37380c8

added score checks mixin to models nexB#1283

5502b5f

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into feat-models_integration

2aeb2a4

empty details in score response handled nexB#1283

103fca0

Signed-off-by: 404-geek <[email protected]>

changed class names to camel case for models and modified the tests n…

952e6a6

…exB#1283 Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

3ba3db5

Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

2f2b846

Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

39a056d

Signed-off-by: 404-geek <[email protected]>

docstrings formatted nexB#1283

e5f3e7a

Signed-off-by: 404-geek <[email protected]>

created basic fetch and availability functions for scorecode pipeline…

c2f5c4d

… nexB#598 Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

6ee5b07

Merge remote-tracking branch 'origin/scorecard_integration' into scor…

760afc2

…ecard_integration

modified doc strings and models and imported ScoreCode package in set…

0dbc92f

…up.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]>

setup.cfg nexB#1283

d652f42

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

aa154c0

Merge branch 'refs/heads/feat-models_integration' into scorecard_inte…

37ad73a

…gration # Conflicts: # scanpipe/models.py # scanpipe/tests/test_models.py

reinstated deleted code during rebase nexB#1283

563991b

Signed-off-by: 404-geek <[email protected]>

code formatting nexB#1283

4632dfc

Signed-off-by: 404-geek <[email protected]>

database migrations for scorecard nexB#1283

923c834

Signed-off-by: 404-geek <[email protected]>

updated the scanpipe only fields nexB#1283

94bfcd0

Signed-off-by: 404-geek <[email protected]>

changed scorecode commit hash for latest pull nexB#1283

d129f73

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

24c7be0

update pipeline code and changed scorecode hash commit nexB#1283

259f004

Signed-off-by: 404-geek <[email protected]>

404-geek added 4 commits August 20, 2024 15:48

Merge branch 'main' into scorecard_integration

88a43dc

moved the data to be regenerated if reqiured nexB#1283

fc4945e

Signed-off-by: 404-geek <[email protected]>

updated urls for testing nexB#1283

c00b3b3

Signed-off-by: 404-geek <[email protected]>

added merged migration file nexB#1283

b97ff7a

Signed-off-by: 404-geek <[email protected]>

tdruez requested changes Aug 21, 2024

View reviewed changes

404-geek added 5 commits August 23, 2024 01:02

Changed docstring and renamed functions according to suggestions nexB…

496945b

…#1283 Signed-off-by: 404-geek <[email protected]>

class name changes in steps of pipeline nexB#1283

f86d5bb

Signed-off-by: 404-geek <[email protected]>

pipeline name updated nexB#1283

36e955a

Signed-off-by: 404-geek <[email protected]>

Merge branch 'aboutcode-org:main' into scorecard_integration

90c113a

update pipeline code and steps nexB#1283

43886a3

Signed-off-by: 404-geek <[email protected]>

404-geek added 10 commits October 29, 2024 21:20

Merge branch 'main' into scorecard_integration

7c134c7

update pipeline steps to work with scorecode 0.0.2 release nexB#1283

3d4d6ea

Signed-off-by: 404-geek <[email protected]>

Merge remote-tracking branch 'origin/scorecard_integration' into scor…

f4ed4b5

…ecard_integration

update migration nexB#1283

b9229a5

Signed-off-by: 404-geek <[email protected]>

Merge branch 'aboutcode-org:main' into scorecard_integration

f80f111

update migration nexB#1283

ee2ea14

Signed-off-by: 404-geek <[email protected]>

rename pipeline name with data parsing function nexB#1283

bd5b9b3

Signed-off-by: 404-geek <[email protected]>

code valid nexB#1283

2b4629e

Signed-off-by: 404-geek <[email protected]>

update setup.cfg nexB#1283

99ee48d

Signed-off-by: 404-geek <[email protected]>

optimize code while saving score checks nexB#1283

80153f8

Signed-off-by: 404-geek <[email protected]>

404-geek requested review from tdruez and AyanSinhaMahapatra December 2, 2024 22:13

404-geek added 4 commits December 2, 2024 17:49

Merge branch 'aboutcode-org:main' into scorecard_integration

77efae0

update test cases and regen scorecard data logic nexB#1283

113a557

Signed-off-by: 404-geek <[email protected]>

update migration file nexB#1283

64e17fe

Signed-off-by: 404-geek <[email protected]>

remove unwanted change nexB#1283

027fe2d

Signed-off-by: 404-geek <[email protected]>

tdruez requested changes Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scorecard Integration #1294

Scorecard Integration #1294

404-geek commented Jun 26, 2024 •

edited

Loading

tdruez Aug 21, 2024

tdruez Aug 21, 2024

pombredanne commented Sep 29, 2024

tdruez commented Oct 29, 2024

404-geek commented Oct 29, 2024

tdruez left a comment •

edited

Loading

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024

tdruez Dec 3, 2024



		class ScorecardCheck(UUIDPKModel, ScorecardChecksMixin):
		def __str__(self):

Scorecard Integration #1294

Are you sure you want to change the base?

Scorecard Integration #1294

Conversation

404-geek commented Jun 26, 2024 • edited Loading

ScoreCode Integration

Key Features:

Related Issues:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pombredanne commented Sep 29, 2024

tdruez commented Oct 29, 2024

404-geek commented Oct 29, 2024

tdruez left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

404-geek commented Jun 26, 2024 •

edited

Loading

tdruez left a comment •

edited

Loading