Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror

Comment Re:Is this bad? (Score 1) 240

The non-profit controlling the pool can negotiate the user fees based on usage. As for people who bypass the pooled crawler -- every web site should let them make requests and then never respond to those request effectively keeping them in infinite timeouts. Public embarrassment of the bypassing entities will also help control this.

Comment Re:Is this bad? (Score 1) 240

There is a good solution to prevent gaming this. The crawler can use AI to assess whether it wants to pay the price the page is asking or not. It can always decide the price is to high and not add the page to the index. In that case it doesn't pay. The payment is not for crawling, it is for giving permission to be added to the global index.

Comment Re:Is this bad? (Score 1) 240

Another option would be to let each page set a micro payment amount in it's headers. Then the crawler could crawl until they run out of money. This works as a double-edged sword. If you set your micro payment amount too high you are not going to get crawled and then you'll drop out of every search index. So it's your choice. The single crawler would crawl free pages first and then crawl from cheapest to most expensive until it runs out of money. Obviously if you set your micro payment at $100 you're never going to get crawled and you'll never appear in another search engine either.

Comment Re:Is this bad? (Score 1) 240

I would like to see the Google anti-trust trial solve this. A good solution is to have a single crawler for the web (can be Google's via anti-trust settlement) and then everyone pays into a pool to get access to the feeds from that single crawler. Payment into that pool can then be used to make the equivalent of statutory royalty payments to the sites crawled. If you don't want to be crawled put your stuff behind a login. Of course you are going to be sorely disappointed in the amount you get from those statutory payments simply because of the sheer number of web pages (around 47B). Optimistically you might be looking at $0.10 a page/yr.

Comment Re:I know what they mean (Score 1) 101

It is not at all deterministic. That's because all of the tools I am using are under constant development. If everything in my environment stopped changing I suspect it would be deterministic, but that's never going to happen.

Hallucination is somewhat under your control there are three main sources 1) you exceed the context window of the AI model. For that one you just need to learn the limits of your model and not exceed them. Don't ask it to do a refactor which is going to touch two million lines of code. Two million lines of code won't fit into the context window so it's going to hallucinate substitutions for what doesn't fit. 2) It will hallucinate when your prompt is not specific enough and it fills in the blanks on its own. You can also control that by making very specific prompts. 3) Sometimes it just goes off the rails. For that one you have to closely monitor the terminal tracking what the agent is doing and if it goes off into the weeds, stop it and explain how to get back on track. That's similar to what you need to do with junior programmers.

I would say I am not ending up with any hallucinations (that I am aware of) in the code I am generating, but.. that's because I am very closely watching everything it is doing and I test and review everything manually before committing. If you're a vibe coder who gives a prompt and then goes off to lunch while it works, you're going to have problems and end up throwing away massive amounts of generated code. The second you stop reviewing what it is doing, you are doomed because you will accumulate piles of code you don't understand and can't fix.

Comment Re: If you have a mediocre workforce at best (Score 1) 101

What turns a junior programmer into a senior on is experience dealing with problems of ever increasing complexity. Currently there is an upper bound to the complexity the model can handle; I'd estimate it at around a 25 year old programmer. It is going to take AGI to replace the senior developers.

Consider that I have been feeding the AI tasks like this for a month now on this project. I am using it to write a complete Android app. It is up to 752 files and 110,000 lines of code. Looking at my history I can see that I have given it 30-50KB prompts about 35 times. That sort of implies that a 1.5MB prompt is needed to get to where I am at currently. So where would that 1.5MB prompt come from? That's the part it can't do yet. I'd have to spend a couple weeks working with the AI on that prompt which is a far more manual process than watching it execute the prompt. Then there is the part where I have to continuously supervise it or it will wander off on a tangent. The key bit here is, I know what a tangent is, and it doesn't.

So I suspect it is going to happen, but it is going to take AGI. The task of taking input from a VP and marketing and turning it into a product is far too unstructured and nebulous for today's AIs.

Comment Re:If you have a mediocre workforce at best (Score 1) 101

Around 37 minutes in it got lost and hallucinated. I had to stop it and explain reality to it.

At 47 minutes it got all of the tests passing. 3,062 lines of code changed. A little over 200 agentic steps.

Quick run-time test. It missed hooking the new code up to the UI so I can't trigger it. I tell it and it figures out it forgot to implement an entire screen. That will take another 15 minutes. So I will leave it running and go to bed. It is unlikely to do major damage at this stage of the process. I can see it has already generated 500 lines of code building the missing screen.

Tomorrow I will need to spend an hour manually testing and inspecting the code, then it should be good to commit. So about two hours to replace three or four days.

Slashdot Top Deals

It has just been discovered that research causes cancer in rats.

Working...