Can someone help me break down this code?

Hello! While developing my conventional search engine, I’ve stumbled across this really neat program called Videoshack. Its a video search engine that gets videos via a webscraper, which I can combine with my search engine to make it more professional, like google.

The only problem is that I can barely understand how it works. Specifically, when it comes to scraping and displaying videos. I’ve asked around other forums, and apparently what I’ve trying to do is so niche that I’ll be unable to find any tutorials on it. Though I did learn a few things about the process.

Firstly, I have to implement the API into videoshack. This is a bit of a hurdle in itself since it seems like some APIs must be implemented differently when compared to other API. Next, Videoshack must call on that API, to scrape videos from it, and then it can display those videos on the web page. Videoshack seems to use both Javascript and Json to make this happen, but learning how to do it shouldn’t be a problem for me if I had a little more guidance.

My issues are as followed:

1.) How can I implement an API with Videoshack for scraping? I’ve been looking at the documentation for Pornhub’s API (lmao), but it looks like the instructions to implement the API are completely different than the way Videoshack does it. Does this mean I’m supposed to combine the two, or what? An example for the API implementation is below.

2.) How does it then display the videos? I’m more used to displaying search topics with a database, but Videoshack doesn’t seem to come with one.

Typically, I stay away from asking question like this, but there’s just no documentation for what I’m trying to do here, so I have no choice.

Here’s some links for the API implementation that I talked about

I believe this is how the PH scraper gets webpages: const utils = require('./utils');const utils_scrap = require('./helpers/utils_ - Pastebin.com

(I don’t know why the link seems weird; I can’t fix it).

And here’s how Video’s hack scrapes for videos: case "vimeo": got("https://vimeo.com/search?q=" + encodeURI - Pastebin.com

Everything seems to be implemented in different ways, and I’m not sure how I’d learn to do it myself. Any ideas?

Two different tools = two different APIs.

I’m not sure I understand the question. If you want your app (your SE) to use Videoshack you’ll have to learn how the latter’s API works, and use it accordingly, not the other way around. Videoshack is an archived project, it’s TODO list is not complete and will never be, unless someone else takes it on from where it left. So chances are there’ll be no instructions either.

Yes, study the source code of Videoshack, and then take on its development yourself, this way you’ll be able to better integrate it into your search engine by tweaking it according to need.

Ah, I see. Hmm, I might be out of my depth then here. Still, I think the longer I look at it, the more I understand so I don’t want to throw in the towel just yet. The worst part is that there’s no real tutorials for something like this. Well, there’s some but they’re out dated or in different coding languages. I guess I’ll have to nut up and get to learning one way or another.

Thanks for the answer!

Hence my advise to take on development of Videoshack, it would be a good way to learn while doing, and its better than starting from scratch. Since there are so few libraries that do what you need, is better to take on an archived project than wait for a newer one to come in. I’m confident that once you get to know the code all things will fall in their place (also, chances are that the code might be well commented too).

Well yeah, I guess giving up would just be stupid. As I said in the above, I’m close to creating a search engine that’s almost exactly like google’s in the way it works. Right now I can search for links, and if I got videoshack to work with it, I’d have the video search function working too. I’d be dumb to stop now.

My biggest issue is the fact that a lot of the code is too hard for me to understand right now. I work best when I can see a practical example on how to implement something, but since this project is so niche, I don’t have that. I do have the end result of the implementation, so maybe I can work off of that.

I think that the harder part in creating a high quality video search engine would be coming up with right algorithms — to parse videos metadata, to process images, and of course to store all this data in an efficient way (time/space), plus algorithms for smart keyword matching (fuzzy matching for spellings) and language auto-detection, etc.

There’s a lot of high-level math there, something that is challenging for most engineers.

2 Likes