Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

help wanted - quickstart / examples that work #20

Open
heepspray opened this issue Oct 21, 2018 · 18 comments
Open

help wanted - quickstart / examples that work #20

heepspray opened this issue Oct 21, 2018 · 18 comments

Comments

@heepspray
Copy link

Hi guys

I am Looking for a way to automatically import scan files

I have followed the quickstart on https://github.com/aaronweaver/defectdojo_api/tree/master/examples and I am using the dojo_ci_cd.py with scan files that are imported perfectly when using the GUI. It does not work. I have tried every advice on here, including using defectdojo_api==1.1.3

Am I using the wrong tool or method ? Is there a prerequisite that I don't know of ? Am I just stupid, or is the API broken ? Can someone help me with a basic setup to make this work ?

I have posted in a number of issues on here, but there seems to be no resolution or updates in a good while, in particular:

Any feedback appreciated
@heepspray heepspray changed the title help wanted - help wanted - quickstart / examples that work Oct 21, 2018
@MaroonBeret
Copy link

@aaronweaver having the same problem here, mind looking into this? Thanks

@MaroonBeret
Copy link

@heepspray been able to fix the problem? You could use curl to post a report as well. Btw, Retire.js seems not to be supported by Dojo anymore, that might be your problem.

@heepspray
Copy link
Author

heepspray commented Oct 29, 2018 •
edited
Loading

Hey @MaroonBeret

No, I am still stuck. The amount of errors I ran into while trying made me believe that either a) the wrapper is broken OR b) Everything I do is wrong and I need help

All the issues are stale, so either the users have quietly resolved their problems, or they are using a different method for uploading

So.. are you saying I should just drop the wrapper, and shoot the upload straight at the API ? Does that work for you ?

By the way, should I use API v1 or v2 ?

And what do you mean Retire.js may be a problem ? I only know that as a burp extension..

Thanks for the reply :) sorry about all the questions

@aaronweaver
Copy link
Contributor

aaronweaver commented Oct 29, 2018 via email

@MaroonBeret
Copy link

@heepspray okay gotcha, I've found a way to report my scan results and I'm gonna try to help you.
No problem at all btw, I ran into a lot of issues as well :P.

It will sound weird but fill in all user information for the user you use to post the scan. This includes first name, last name, email address, etc. Dojo needs this info to correctly link the uploaded scan to the uploading user.

If this doesn't work you could try posting with curl. Let me know if thats the case.

@aaronweaver
Copy link
Contributor

aaronweaver commented Oct 30, 2018 via email

@MaroonBeret
Copy link

@aaronweaver Nice, it's appreciated!

@heepspray
Copy link
Author

Thanks for trying to help guys, I did another concentrated attempt and will try to explain :

My main struggle is dojo_ci_cd.py - the file does not run and there is no request sent to the server unless I make a series of horrible hacks in it. I will not be posting the web request here before I get to a point where I can confidently run the example script

The example on https://github.com/aaronweaver/defectdojo_api/tree/master/examples reads :
dojo_ci_cd.py --product=1 --file "/tests/scans/Bodgeit-burp.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=<api_key> --user=admin

The command I am using is :
python dojo_ci_cd.py --product=10 --file "/xx/xx/xx/burp_ad/a9.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=xxxxxxxxxxxx --user=xxxxxxxxx

(--product is an actual product id on my server. And the scan file is a working file that can be imported through the UI)

When I run the file it results in the following failure :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 145, in processFiles
    if test_id.success == False:

On line 86 in dojo_ci_cd.py I change scannerName from None to "Burp Scan". I have no clue why scannerName is set to None in the first place. When I run the file again I get the following error :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 146, in processFiles
    print "Upload failed: Detailed error message: " + test_id.data
TypeError: cannot concatenate 'str' and 'NoneType' objects

I can keep hacking the dojo_ci_cd.py file, but it is obvious that what I am doing is wrong. And it will probably worsen the more I try, so I will stop right here :)

@MaroonBeret
Copy link

@heepspray You might wanna use curl to post your request instead of the Python script, like this:

curl --fail
--header "Authorization: ApiKey $DEFECT_DOJO_API_KEY"
--form "file=@<REPORT_FILE>"
--form "scan_type=<SCAN_TYPE>"
--form "engagement=/api/v1/engagements/<ENGAGEMENT_ID>/" \
--form "active=true"
--form "verified=False"
--form "scan_date=$DATE"
$DEFECT_DOJO_DOMAIN/api/v1/importscan/"

@heepspray
Copy link
Author

@MaroonBeret YES! - thank you :) that works just fine

I see no problem in using this going forward, not sure what to do with the original Issue though.. if you believe that all my problems have been caused by my misunderstandings and general incompetence then it can probably be closed :/

Thanks again, see you on the next hurdle ;)

@MaroonBeret
Copy link

@heepspray Good to know that it worked! As I said, this doesn't mean you are incompetent, I had the same issue and someone else helped me out as well. Doesn't mean this issue doesn't persist though, we just use another approach now. Good luck!

@ylorgat
Copy link

ylorgat commented Nov 14, 2018 •
edited
Loading

@MaroonBeret I'm having issues using the curl method could you paste an example on how to use it correctly.

My scan files are in Downloads/sample-scan-files and I'm using zap

Thank you very much

@heepspray
Copy link
Author

@ylorgat this is a working example :

curl --fail --header "Authorization: ApiKey xxxxxxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --form "file=@/path/to/report/report.xml" --form "scan_type=ZAP Scan" --form "engagement=/api/v1/engagements/123/" --form "active=true" --form "verified=False" --form "scan_date=2018-11-14" example.com/api/v1/importscan/ --verbose

@valentijnscholten
Copy link
Member

I think some of the problems may be caused by the pip version lagging behind what is on github, see #22

@wheelq
Copy link

wheelq commented May 20, 2019 •
edited
Loading

Thanks for trying to help guys, I did another concentrated attempt and will try to explain :

My main struggle is dojo_ci_cd.py - the file does not run and there is no request sent to the server unless I make a series of horrible hacks in it. I will not be posting the web request here before I get to a point where I can confidently run the example script

The example on https://github.com/aaronweaver/defectdojo_api/tree/master/examples reads :
dojo_ci_cd.py --product=1 --file "/tests/scans/Bodgeit-burp.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=<api_key> --user=admin

The command I am using is :
python dojo_ci_cd.py --product=10 --file "/xx/xx/xx/burp_ad/a9.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=xxxxxxxxxxxx --user=xxxxxxxxx

(--product is an actual product id on my server. And the scan file is a working file that can be imported through the UI)

When I run the file it results in the following failure :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 145, in processFiles
    if test_id.success == False:

On line 86 in dojo_ci_cd.py I change scannerName from None to "Burp Scan". I have no clue why scannerName is set to None in the first place. When I run the file again I get the following error :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 146, in processFiles
    print "Upload failed: Detailed error message: " + test_id.data
TypeError: cannot concatenate 'str' and 'NoneType' objects

I can keep hacking the dojo_ci_cd.py file, but it is obvious that what I am doing is wrong. And it will probably worsen the more I try, so I will stop right here :)

After hardcoding those values I get:

Upload failed: Detailed error message:
==============================================
Traceback (most recent call last):
  File "dd_import.py", line 239, in <module>
    class Main:
  File "dd_import.py", line 288, in Main
    summary(dd, engagement_id, test_ids, max_critical, max_high, max_medium)
  File "dd_import.py", line 164, in summary
    print "Total Number of Vulnerabilities: " + str(findings.data["meta"]["total_count"])
TypeError: string indices must be integers

@wurstbrot
Copy link
Contributor

This works (based on securecodebox): https://github.com/wurstbrot/defectdojo-java-api
It will be moved to DefectDojos repo space as soon as @aaronweaver will create a repo in that space.

cURL/Bash-based works also, but is no longer maintained: https://github.com/wurstbrot/defectdojo_api

@dougmcdorman
Copy link

Not using the Python API, but it is a Python example of uploading a scan.

from defectdojo_api import defectdojo
from datetime import datetime, timedelta
import os, sys
import argparse
import time
import requests

host = 'http://localhost:8080'
authn_value = 'Token aaaaabbbbbcccccddddddddeeeeeeeffffffff'

filename = '/home/username/zap-report.xml'

headers = {'Authorization': authn_value}
base_url = host
url = base_url + '/api/v2/import-scan/'

engagement_id = '7'
scan_type = 'ZAP Scan'
lead = 'admin'
active = 'true'
verified = 'true'
close_old_findings = 'true'
skip_duplicates = 'true'
scan_date = '2019-09-04'
tags = ''
build_id = ''
minimum_severity = ''
with open(filename, 'rb') as filehandle:
              filedata = filehandle.read()
              data = {
                               'file': filedata,
                               'engagement': ('', engagement_id),
                               'scan_type': ('', scan_type),
                               'active': ('', active),
                               'verified': ('', verified),
                               'close_old_findings': ('', close_old_findings),
                               'skip_duplicates': ('', skip_duplicates),
                               'scan_date': ('', scan_date),
                               #'tags': ('', tags), # tags gets upset if it is empty
                               'build_id': ('', build_id),
                               'minimum_severity': ('', minimum_severity)
                           }
              my_request = requests.Session()
              response = my_request.post(url=url, files=data, headers=headers)
              print(response.text)
              print(response.status_code)
print(requests.status_codes._codes[response.status_code])`

@willnil
Copy link

willnil commented Oct 25, 2019

Thanks for trying to help guys, I did another concentrated attempt and will try to explain :
My main struggle is dojo_ci_cd.py - the file does not run and there is no request sent to the server unless I make a series of horrible hacks in it. I will not be posting the web request here before I get to a point where I can confidently run the example script
The example on https://github.com/aaronweaver/defectdojo_api/tree/master/examples reads :
dojo_ci_cd.py --product=1 --file "/tests/scans/Bodgeit-burp.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=<api_key> --user=admin
The command I am using is :
python dojo_ci_cd.py --product=10 --file "/xx/xx/xx/burp_ad/a9.xml" --scanner="Burp Scan" --high=0 --host=http://localhost:8000 --api_key=xxxxxxxxxxxx --user=xxxxxxxxx
(--product is an actual product id on my server. And the scan file is a working file that can be imported through the UI)
When I run the file it results in the following failure :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 145, in processFiles
    if test_id.success == False:

On line 86 in dojo_ci_cd.py I change scannerName from None to "Burp Scan". I have no clue why scannerName is set to None in the first place. When I run the file again I get the following error :

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 146, in processFiles
    print "Upload failed: Detailed error message: " + test_id.data
TypeError: cannot concatenate 'str' and 'NoneType' objects

I can keep hacking the dojo_ci_cd.py file, but it is obvious that what I am doing is wrong. And it will probably worsen the more I try, so I will stop right here :)

After hardcoding those values I get:

Upload failed: Detailed error message:
==============================================
Traceback (most recent call last):
  File "dd_import.py", line 239, in <module>
    class Main:
  File "dd_import.py", line 288, in Main
    summary(dd, engagement_id, test_ids, max_critical, max_high, max_medium)
  File "dd_import.py", line 164, in summary
    print "Total Number of Vulnerabilities: " + str(findings.data["meta"]["total_count"])
TypeError: string indices must be integers

I am having the same TypeError now when using the defectdojo_api example python script: dojo_ci_cd.py trying to import my json scan files. I guess the problem might due to the json parser. I tried to hack the python file, like adding json.loads() or json.dumps() to the data from findings, but did not success.

Traceback (most recent call last):
  File "/home/ubuntu/persistant_storage/defectdojo_api/examples/dojo_ci_cd.py", line 244, in <module>
    class Main:
  File "/home/ubuntu/persistant_storage/defectdojo_api/examples/dojo_ci_cd.py", line 293, in Main
    summary(dd, engagement_id, test_ids, max_critical, max_high, max_medium)
  File "/home/ubuntu/persistant_storage/defectdojo_api/examples/dojo_ci_cd.py", line 168, in summary
    print "Total Number of Vulnerabilities: " + str(findings.data["meta"]["total_count"])
TypeError: string indices must be integers

@heepspray I have the same problem. Did you solve this or u try another approach to import your scans automatically?

Thank you all :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@aaronweaver @wurstbrot @wheelq @valentijnscholten @dougmcdorman @MaroonBeret @willnil @ylorgat @heepspray