-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: auto add dem file if not provided by the user #380
base: develop
Are you sure you want to change the base?
Conversation
@@ -0,0 +1,92 @@ | |||
# Scrapy settings for jaxa project | |||
# |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think, you set up a Scrapy app(Jaxa), which is great! However, I highly recommend using Scrapy's Core API. It simplifies the workflow and is perfect for single-page scripts without requiring a full project setup.
Hereβs an example:
from scrapy.crawler import CrawlerRunner
from scrapy import Spider
from twisted.internet import reactor
class ExampleSpider(Spider):
name = "example"
start_urls = ["https://example.com"]
def parse(self, response):
print("Scraped content:", response.body[:100])
runner = CrawlerRunner()
runner.crawl(ExampleSpider)
runner.join().addBoth(lambda _: reactor.stop())
reactor.run()
6b7ef4c
to
cae4a94
Compare
for more information, see https://pre-commit.ci
304d16a
to
1f0baf5
Compare
for more information, see https://pre-commit.ci
1fa5675
to
850f07a
Compare
Nice π Is the text supposed to say 'approval for lock task'? Also having it marked mandatory doesn't make sense if you provide a default and its a radio. Perhaps it might help adding a small bit of explanation: 'We provide a global elevation model that should be suitable for most cases, but you may wish to upload a more detailed custom model in certain scenarios (for example steep mountainous terrain)' |
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
What type of PR is this? (check all applicable)
Describe this PR
This PR introduces a feature where if the Project creator wants DEM file to be automatically uploaded instead of him uploading the file. Scrapy Bot will crawl the "Jaxa" website for the dem file and if there are multiple images that falls in that selected range then it merges all into one DEM file and later uploaded to S3 bucket.
Review Guide
Notes for the reviewer. How to test this change?
Checklist before requesting a review