+ 
+ 
+ 
@@ -209,7 +236,7 @@ 
@@ -237,7 +264,7 @@ 
@@ -267,7 +294,7 @@ 
diff --git a/posts/posts_8.html b/posts/posts_8.html
new file mode 100644
index 0000000..41a2c7a
--- /dev/null
+++ b/posts/posts_8.html
@@ -0,0 +1,455 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Overcoming Security Challenges and Extracting Comprehensive Data from Walmart.com
+
+ Web scraping offers immense potential for data collection, insights, and analytics, but sites + like Walmart.com have stringent security measures to protect their data. In this blog post, + I’ll delve into the challenges faced, methods employed to bypass advanced security + protocols, and some insights into the data scraped from Walmart.com.
+ +Project Overview
+ +The project aimed to scrape Walmart.com, an e-commerce giant, to gather a range of data, + including product details, pricing, availability, reviews, and seller information. The + complexity of the project stemmed from Walmart's advanced security measures and the need to + circumvent them ethically without disrupting the website’s functionality.
+ +Challenges Faced
+ +As expected, scraping Walmart.com presented unique challenges. Here are the key obstacles and + solutions I used to achieve a successful scrape:
+ +1. IP Blocking
+Walmart.com uses IP blocking to prevent continuous or suspicious access from single IPs. This + measure limits how frequently one can make requests, as high-frequency requests from the + same IP address often signal a bot or web scraper.
+Solution: To avoid triggering IP blocks, I implemented a rotating proxy + system. By using multiple proxies, I was able to distribute requests across different IP + addresses, thereby mimicking human-like behavior and avoiding detection. Additionally, I + added randomized delays between requests to further reduce the likelihood of being flagged. +
+ +2. "Press & Hold" CAPTCHA
+CAPTCHA tests are designed to distinguish between humans and bots, and Walmart.com employs + the "Press & Hold" CAPTCHA to prevent automated access.
+Solution: Solving this CAPTCHA type involved emulating real user behavior + with automated tools. I adjusted the request headers and user-agent strings to closely match + those of legitimate browsers, making it less likely to trigger CAPTCHA tests. For instances + where CAPTCHAs still appeared, I employed image recognition techniques and manual + CAPTCHA-solving services, ensuring the scraping process continued without interruption.
+ +3. Advanced Security Measures: PerimeterX and Akamai Bot Manager
+Walmart.com deploys PerimeterX and Akamai Bot Manager to detect and block bots by analyzing + behavioral patterns, request headers, and browsing fingerprints.
+Solution: Bypassing these systems required precision. I carefully configured + each request’s headers to include randomized elements (such as user-agent, language + settings, and accepted content types) that closely mimic a human browsing session. + Additionally, I utilized browser automation tools to simulate authentic browsing behavior, + including mouse movements, scrolls, and clicks, ensuring the requests appeared as though + they were made by an actual user.
+ +Data Collected
+ +With these solutions in place, I was able to scrape a comprehensive dataset from Walmart.com, + including detailed product information. Below is an example of the data fields collected: +
+ +
+{
+ "ProductId": "2UW4JTE2A5UZ",
+ "ProductUrl": "https://www.walmart.com/ip/Men-s-1-10-ctw-Black-Diamond-Black-Tungsten-Grooved-8MM-Wedding-Band-Men-s-Ring/949552322?classType=VARIANT",
+ "ProductName": "Brilliance Fine Jewelry Men's 1/10 Ctw Black Diamond Black Tungsten Grooved 8MM Wedding Band - Men's Ring",
+ "category": "Jewelry",
+ "ProductPrice": "$98",
+ "wasPrice": "",
+ "savingsAmount": "",
+ "sellerName": "Walmart.com",
+ "availabilityStatus": "In stock",
+ "rating": 4.4,
+ "numberOfReviews": 112,
+ "brand": "Brilliance Fine Jewelry",
+ "manufacturerProductId": "TG17204",
+ "returnPolicy": {
+ "returnable": true,
+ "freeReturns": true,
+ "returnWindow": {
+ "value": 90,
+ "unitType": "Day"
+ },
+ "returnPolicyText": "Free Holiday returns until Jan 31",
+ "returnPolicyTextCode": {
+ "code": "PRODUCT_HOLIDAY_RETURN",
+ "data": null
+ },
+ "returnPolicyCondition": null,
+ "holidayReturnEnabled": true
+ },
+ "description": "Bold and elegant black diamonds make a striking statement in this black ion-plated tungsten men's wedding ring. Step away from traditional wedding band choices and select a unique ring that truly reflects your style and personality. This modern design is perfect for men of all ages\u2014whether you\u2019re a teenager looking to make your mark, a young adult ready to commit, or a senior celebrating a lifetime of love. The eye-catching black diamonds are complemented by the sleek tungsten band, creating a distinctive look that stands out on any occasion. Embrace your individuality with this stylish black ion-plated tungsten ring, a perfect choice for anyone seeking a contemporary and sophisticated wedding band.",
+ "ProductImages": [
+ {
+ "url": "https://i5.walmartimages.com/seo/Men-s-1-10-ctw-Black-Diamond-Black-Tungsten-Grooved-8MM-Wedding-Band-Men-s-Ring_1b8b58e3-3119-4020-a8c1-1bcc4a8c7da0.c487b34ddd2267ec0018e877b2308a83.jpeg"
+ },
+ {
+ "url": "https://i5.walmartimages.com/asr/10fe3fca-a156-43de-8071-6172d3420909.9764ef731c98a7230689f8ead754e6c2.jpeg"
+ },
+ {
+ "url": "https://i5.walmartimages.com/asr/0cd08853-3632-4c78-8953-2f0c0d6a4223.18f699126bb6fc945be679dcd56dc442.jpeg"
+ },
+ {
+ "url": "https://i5.walmartimages.com/asr/4c26bec3-7742-44fa-8d47-88672992c112.2d9efab43a6c248b0d6f5ae8be6a5197.jpeg"
+ }
+ ],
+ "reviews": {
+ "customerReviews": [
+ {
+ "reviewId": "360058301",
+ "rating": 5,
+ "reviewSubmissionTime": "10/12/2024",
+ "reviewText": "This ring is amazing quality for the price. My husband loves it. He never takes it off and it looks the same way it did when I purchased it.",
+ "reviewTitle": "Great quality!",
+ "negativeFeedback": 0,
+ "positiveFeedback": 0,
+ "userNickname": "Sav",
+ "fulfilledBy": "Walmart",
+ "status": null,
+ "sellerName": "Walmart.com",
+ "media": null,
+ "photos": [],
+ "badges": [
+ {
+ "badgeType": "Custom",
+ "id": "VerifiedPurchaser",
+ "contentType": "REVIEW",
+ "glassBadge": {
+ "id": "VerifiedPurchaser",
+ "text": "Verified Purchase"
+ }
+ }
+ ],
+ "clientResponses": null,
+ "syndicationSource": null,
+ "snippetFromTitle": null,
+ "features": [
+ {
+ "name": "Ring size",
+ "value": "7"
+ }
+ ]
+ }
+ ],
+ "topNegativeReview": {
+ "reviewId": "359851258",
+ "rating": 2,
+ "reviewSubmissionTime": "10/10/2024",
+ "userNickname": "Laura",
+ "negativeFeedback": 0,
+ "positiveFeedback": 0,
+ "reviewText": "I had this as a gift to my husband. I wrote a special note to come with it and they didn't put it in there so it ruined my vow renewal question ring is perfect but ruined everything that was planned.",
+ "reviewTitle": null,
+ "badges": [
+ {
+ "badgeType": "Custom",
+ "id": "VerifiedPurchaser",
+ "contentType": "REVIEW",
+ "glassBadge": {
+ "id": "VerifiedPurchaser",
+ "text": "Verified Purchase"
+ }
+ }
+ ],
+ "clientResponses": null,
+ "syndicationSource": null,
+ "snippetFromTitle": null,
+ "media": null
+ },
+ "topPositiveReview": {
+ "reviewId": "360058301",
+ "rating": 5,
+ "reviewSubmissionTime": "10/12/2024",
+ "userNickname": "Sav",
+ "negativeFeedback": 0,
+ "positiveFeedback": 0,
+ "reviewText": "This ring is amazing quality for the price. My husband loves it. He never takes it off and it looks the same way it did when I purchased it.",
+ "reviewTitle": "Great quality!",
+ "badges": [
+ {
+ "badgeType": "Custom",
+ "id": "VerifiedPurchaser",
+ "contentType": "REVIEW",
+ "glassBadge": {
+ "id": "VerifiedPurchaser",
+ "text": "Verified Purchase"
+ }
+ }
+ ],
+ "syndicationSource": null,
+ "snippetFromTitle": null,
+ "clientResponses": null,
+ "media": null
+ }
+ }
+}
+Sample Data
+ +A sample dataset is available on my GitHub + repository. The JSON file (walmart.com_sample_data.json) + showcases the + structure and details captured for each product, providing insight into the data points + available for analysis.
+ +Key Insights and Analysis
+ +The data collected from Walmart.com provides extensive information about each product, + enabling rich insights, including:
+-
+
- Pricing Trends: Comparison between current prices, discounts, and the + original price. +
- Consumer Feedback: Rating and review analysis help understand product + reception and common customer concerns. +
- Availability Status: Real-time stock information for timely purchasing + or recommendations. +
Lessons Learned and Takeaways
+ +Through this project, I developed new strategies for overcoming cutting-edge security + measures while remaining compliant with legal and ethical standards. This experience not + only strengthened my expertise in web scraping but also reinforced the importance of + responsible data collection practices.
+ +Conclusion
+ +Scraping websites like Walmart.com is both challenging and rewarding, especially with + advanced security protocols in place. Overcoming IP blocks, CAPTCHAs, and bot managers + required a combination of technical expertise, strategic planning, and ethical + responsibility. The data gathered has significant value in e-commerce analysis and consumer + behavior insights.
+ +🔗 Explore the Project and Connect
+If you’re interested in learning more or have similar projects in mind, check out the full + project on GitHub: walmart.com + Scraper. I’d love to hear your feedback and connect with fellow developers!
+ +For inquiries or service requests, feel free to reach out via LinkedIn or + visit my portfolio + at mominur.dev.
+ ++ Are you ready to leverage the future of data scraping for your business? Contact me today to + explore innovative data solutions that can transform your organization! +
+