Error when Scraping Twitter Data for Isolated Cases

[wieco12]

I tried to scrape Twitter Data with an academic twitter API. The code worked for almost every case, even though there are a few cases, where it doesn’t.

I used the code tweets <- get_all_tweets("@honda lang: en -is:retweet", "2018-08-06T00:00:00Z", "2018-08-26T00:00:00Z", n = Inf)

After scraping 4 pages of tweets, the following error occurred:

“Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token, : Too many errors. In addition: Warning messages: 1: Recommended to specify a data path in order to mitigate data loss when ingesting large amounts of data. 2: Tweets will not be stored as JSONs or as a .rds file and will only be available in local memory if assigned to an object.”

I actually don’t get what the problem is because the code works even for cases with more than 35.000 Tweets. Therefore, I don’t think the number of tweets is the reason.

Can somebody help me?

[brunoanuncia]

Same problem here. When I change the page_n for any value < 500, the API start responding but then stops. Console print below:

> get_all_tweets(query = query,
+                start_tweets = "2021-12-31T21:00:00Z", #(GMT0)
+                end_tweets = "2023-03-08T20:59:59Z", #um mes dos ataques do dia 08/01/23 (GMT0)
+                bearer_token = get_bearer(),
+                n = 10000000,
+                page_n = 100,
+                data_path = "controversia_urnas_20220101-20230208",
+                verbose = TRUE)
query:  urnas OR urna OR (fraude eleitoral) OR (fraude eleição) OR 
(fraude eleições) OR (voto impresso) OR (voto auditável) OR
"sistema eleitoral" lang:pt 
Total pages queried: 1 (tweets captured this page: 98).
Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token,  : 
  Too many errors.
Além disso: Warning messages:
1: Tweets will be bound in local memory as well as stored as JSONs. 
2: Directory already exists. Existing JSON files may be parsed and returned, choose a new path if this is not intended. 

Does anyone know what is happening?