'calulations'=TRUE doesn't return net and percent changes

[SteadyGiant]

When I set 'calculations'=TRUE in the payload, and then use it to call blsAPI(), the response does not contain net or percent changes. The value column contains the same values as it does when I call blsAPI() with 'calculations'=FALSE. See the following:

library(blsAPI)
library(rjson)

payload1 <- list(
  'seriesid'='LAUCN040010000000005',
  'startyear'=2010,
  'endyear'=2012,
  'catalog'=FALSE,
  'calculations'=TRUE,                                                # Here it's TRUE
  'annualaverage'=FALSE,
  'registrationKey'=Sys.getenv("BLS_REG_KEY"))
response1 <- blsAPI(payload1, 2)
json1 = fromJSON(response1)

payload2 <- list(
  'seriesid'='LAUCN040010000000005',
  'startyear'=2010,
  'endyear'=2012,
  'catalog'=FALSE,
  'calculations'=FALSE,                                               # Here it's FALSE
  'annualaverage'=FALSE,
  'registrationKey'=Sys.getenv("BLS_REG_KEY"))
response2 <- blsAPI(payload2, 2)
json2 = fromJSON(response2)

apiDF <- function(data){
  df <- data.frame(year=character(),
                   period=character(),
                   periodName=character(),
                   value=character(),
                   stringsAsFactors=FALSE)
  i <- 0
  for(d in data){
    i <- i + 1
    df[i,] <- unlist(d)
  }
  return(df)
}

unemployed.df1 <- apiDF(json1$Results$series[[1]]$data)
unemployed.df2 <- apiDF(json2$Results$series[[1]]$data)

identical(unemployed.df1, unemployed.df2)
# [1] TRUE 

Am I doing something wrong?

[dmichael555]

Hi, Mike,

I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting.

I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.'

Here's what I put together as a workaround for extracting net changes and percent changes.

#Global cleanup
rm(list = ls(all.names = TRUE))

library(blsAPI)
library(rjson)

my_api_key<-"YOUR_API_KEY_HERE"

#Date Range Parameters

startyear<-2005
endyear<-2021

#CES65000000001, Health and Education Services
payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE)
health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE)
summary(health_and_ed_with_net)
View(health_and_ed_with_net)

test<-fromJSON(health_and_ed_with_net)
View(test)

New Version of loop from original function

#Iterate over the series
number_of_series <- length(test$Results$series)
for (i in 1:number_of_series){
#Set the default structure of the data frame
df_start <- data.frame(year = character(),
period = character(),
periodName = character(),
value = character(),
net_change_1 = character(),
net_change_3 = character(),
net_change_6 = character(),
net_change_12 = character(),
pct_change_1 = character(),
pct_change_3 = character(),
pct_change_6 = character(),
pct_change_12 = character(),
stringsAsFactors = FALSE)
#Get the data
series_data <- test$Results$series[[i]]$data
#Can get no data after a successful request
if (length(series_data) > 0) {
j <- 0 #index at 0 to start
for (d in series_data) {
j <- j + 1 #advance by one
#Remove the footnotes from the list to stop the warnings
d$footnotes <- NULL
d$latest <- NULL
#Add record to the data frame
df_start[j, ] <- unlist(d)
}
#Add in the series id
df_start$seriesID <- test$Results$series[[i]]$seriesID
}
#Create the data frame that will be returned
if (!exists("df_to_return")){
#Data frame to return not defined so create it
df_to_return <- df_start
}
else {
#Append to the existing data frame
df_to_return <- rbind(df_to_return, df_start)
}
}

View(df_to_return)

Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values.

Here's what my test case produced:

It appears to match the lists returned in the JSON response.

Let us know if you've already implemented something similar within the package.

All the best,
Darren

[adhi-r]

Darren, at this point i’d recommend you use the package “blscrapeR”. it’s far better maintained and convenient. no offense intended to mike but i think he’d agree if anything.

…

On Fri, Dec 24, 2021 at 8:22 AM Darren Michael ***@***.***> wrote: Hi, Mike, I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting. I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.' Here's what I put together as a workaround for extracting net changes and percent changes. #Global cleanup rm(list = ls(all.names = TRUE)) library(blsAPI) library(rjson) my_api_key<-"YOUR_API_KEY_HERE" #Date Range Parameters startyear<-2005 endyear<-2021 #CES65000000001, Health and Education Services payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE) health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE) summary(health_and_ed_with_net) View(health_and_ed_with_net) test<-fromJSON(health_and_ed_with_net) View(test) New Version of loop from original function #Iterate over the series number_of_series <- length(test$Results$series) for (i in 1:number_of_series){ #Set the default structure of the data frame df_start <- data.frame(year = character(), period = character(), periodName = character(), value = character(), net_change_1 = character(), net_change_3 = character(), net_change_6 = character(), net_change_12 = character(), pct_change_1 = character(), pct_change_3 = character(), pct_change_6 = character(), pct_change_12 = character(), stringsAsFactors = FALSE) #Get the data series_data <- test$Results$series[[i]]$data #Can get no data after a successful request if (length(series_data) > 0) { j <- 0 #index at 0 to start for (d in series_data) { j <- j + 1 #advance by one #Remove the footnotes from the list to stop the warnings d$footnotes <- NULL d$latest <- NULL #Add record to the data frame df_start[j, ] <- unlist(d) } #Add in the series id df_start$seriesID <- test$Results$series[[i]]$seriesID } #Create the data frame that will be returned if (!exists("df_to_return")){ #Data frame to return not defined so create it df_to_return <- df_start } else { #Append to the existing data frame df_to_return <- rbind(df_to_return, df_start) } } View(df_to_return) Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values. Here's what my test case produced: [image: image] <https://user-images.githubusercontent.com/45790848/147355030-8b214713-74c3-4d99-b195-44c0a74df943.png> It appears to match the lists returned in the JSON response. [image: image] <https://user-images.githubusercontent.com/45790848/147355159-3f5105e0-eb35-490a-b9ad-146fc98c1c19.png> Let us know if you've already implemented something similar within the package. All the best, Darren — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJ4HD66U24VWX2PUUJHHN3USR627ANCNFSM4GPTVAEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dmichael555]

Hi, Adhi, Thanks for the advice. I'll have at the blscrapeR package. All the best to both you and Mike. Regards,Darren On Friday, December 24, 2021, 11:18:43 AM MST, Adhi Rajaprabhakaran ***@***.***> wrote: Darren, at this point i’d recommend you use the package “blscrapeR”. it’s far better maintained and convenient. no offense intended to mike but i think he’d agree if anything.

On Fri, Dec 24, 2021 at 8:22 AM Darren Michael ***@***.***> wrote: Hi, Mike, I know the original post is from a while ago, but I saw the same performance as described in the original post when I used the 'calculations=TRUE' setting. I looked into your code and noticed that the post-processing of the JSON response seems to pull out only four of the columns returned in the response, i.e., year, period, periodName, and value. For the first data element in the returned list, the code assigns NULL to remove 'footnotes' and 'latest.' Here's what I put together as a workaround for extracting net changes and percent changes. #Global cleanup rm(list = ls(all.names = TRUE)) library(blsAPI) library(rjson) my_api_key<-"YOUR_API_KEY_HERE" #Date Range Parameters startyear<-2005 endyear<-2021 #CES65000000001, Health and Education Services payload_net<-list('seriesid'='CES6500000001','registrationKey'=my_api_key,'startyear'=startyear,'endyear'=endyear,'calculations'=TRUE,'annualaverage'=FALSE) health_and_ed_with_net<-blsAPI(payload_net,api_version=2,return_data_frame = FALSE) summary(health_and_ed_with_net) View(health_and_ed_with_net) test<-fromJSON(health_and_ed_with_net) View(test) New Version of loop from original function #Iterate over the series number_of_series <- length(test$Results$series) for (i in 1:number_of_series){ #Set the default structure of the data frame df_start <- data.frame(year = character(), period = character(), periodName = character(), value = character(), net_change_1 = character(), net_change_3 = character(), net_change_6 = character(), net_change_12 = character(), pct_change_1 = character(), pct_change_3 = character(), pct_change_6 = character(), pct_change_12 = character(), stringsAsFactors = FALSE) #Get the data series_data <- test$Results$series[[i]]$data #Can get no data after a successful request if (length(series_data) > 0) { j <- 0 #index at 0 to start for (d in series_data) { j <- j + 1 #advance by one #Remove the footnotes from the list to stop the warnings d$footnotes <- NULL d$latest <- NULL #Add record to the data frame df_start[j, ] <- unlist(d) } #Add in the series id df_start$seriesID <- test$Results$series[[i]]$seriesID } #Create the data frame that will be returned if (!exists("df_to_return")){ #Data frame to return not defined so create it df_to_return <- df_start } else { #Append to the existing data frame df_to_return <- rbind(df_to_return, df_start) } } View(df_to_return) Using this approach, I was able to pull out the remaining data in the list, i.e., the net change and percentage change values. Here's what my test case produced: [image: image] <https://user-images.githubusercontent.com/45790848/147355030-8b214713-74c3-4d99-b195-44c0a74df943.png> It appears to match the lists returned in the JSON response. [image: image] <https://user-images.githubusercontent.com/45790848/147355159-3f5105e0-eb35-490a-b9ad-146fc98c1c19.png> Let us know if you've already implemented something similar within the package. All the best, Darren — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJ4HD66U24VWX2PUUJHHN3USR627ANCNFSM4GPTVAEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: ***@***.***>