Daily master index is not availbale for this date

[fschiro]

Did this function break?

Even the example from the publication doesn't work:

getDailyMaster('08/09/2016')
The daily master index is not available for this date.

Above from publication: edgar: An R Package for the U.S. SECEDGAR Retrieval and Parsing ofCorporate Filings

I looked at the function getDailyMaster and it seems to be navigating to this link: https://www.sec.gov/Archives/edgar/daily-index/
But getDailyMaster seems to be going to a file at that link that is not there. I think they might have changed the file structure?

[fschiro]

It looks like the date range for which link to use is broken. If I force it to use link2 for a 2018 link then the function below works.

GetDailyInfo <- function(day, month, year) {
        dmethod <- getdownCompat()
        date <- paste0(year, month, day)
        filename <- paste0("Daily Indexes/daily_idx_", date)
        link1 <- paste0("https://www.sec.gov/Archives/edgar/daily-index/master.", 
            date, ".idx")
        link1 <- paste0("https://www.sec.gov/Archives/edgar/daily-index/", 
            year, "/QTR", ceiling(as.integer(month)/3), "/master.", 
            date, ".idx")
        link3 <- paste0("https://www.sec.gov/Archives/edgar/daily-index/", 
            year, "/QTR", ceiling(as.integer(month)/3), "/master.", 
            substr(as.character(year), 3, 4), month, day, ".idx")
        link4 <- paste0("https://www.sec.gov/Archives/edgar/daily-index/", 
            year, "/QTR", ceiling(as.integer(month)/3), "/master.", 
            date, ".idx")
        down.success = FALSE
        if (year < 1999) {
            fun.return3 <- DownloadSECFile(link3, filename, dmethod)
            if (fun.return3 == 1) {
                down.success = TRUE
            }
        }
        if (year > 1998 && year < 2012) {
            fun.return4 <- DownloadSECFile(link4, filename, dmethod)
            if (fun.return4 == 1) {
                down.success = TRUE
            }
        }
        if (year > 2011) {
            fun.return1 <- DownloadSECFile(link1, filename, dmethod)
            if (fun.return1 && file.size(filename) > 500) {
                down.success = TRUE
            }
            else {
                fun.return2 <- DownloadSECFile(link2, filename, 
                  dmethod)
                if (fun.return2) {
                  down.success = TRUE
                }
            }
        }
        if (down.success) {
            temp.data <- gsub("'", "", readLines(filename))
            writeLines(temp.data, filename)
            header.end <- grep("--------------------------------------------------------", 
                temp.data)
            scrapped.data <- scan(filename, what = list("", "", 
                "", "", ""), flush = F, skip = header.end, sep = "|", 
                quiet = T)
            final.data <- data.frame(cik = scrapped.data[[1]], 
                company.name = scrapped.data[[2]], form.type = scrapped.data[[3]], 
                date.filed = scrapped.data[[4]], edgar.link = scrapped.data[[5]])
            final.data$edgar.link <- NULL
            file.remove(filename)
            save(final.data, file = paste0(filename, ".Rda"))
            return(final.data)
        }
        else {
            cat(" Daily master index is not availbale for this date.")
            return()
        }
    }