this post was submitted on 15 Jul 2023
505 points (95.8% liked)
Technology
59669 readers
3109 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Nonsense. Less people are using it because there are viable alternatives and the broader novelty has worn off.
I use it every day in my job and the quality of answers only drops off when prompts are poorly crafted.
By and large, the average user doesn’t understand the fundamentals of prompt engineering.
The suggestion that “answers are increasingly dumber” is embarrassing.
Unfortunately I don't agree with you. Different things have changed over time:
All these changes have made working with gpt less pleasant, and more difficult for very advanced and specialized case, particularly with gpt-4 which at the beginning was particularly good.
This was really enlightening. Do you have some articles that elaborate? ☺️
Regarding 3.5 turbo you can check the documentation, the old 3.5 models are defined as "legacy". Regarding max number of tokens of gpt-4 you can try yourself. It used to be >8k, it is now >4k from webui.
There is a talk from openai cio (if I recall correctly) where he describes that reinforcement learning from human feedback (rlhf) actually decreased performance of the models when it comes to programming. I cannot find it now, but it is around on YouTube.
The additional safeguard against jailbreaking, it is what OpenAI has been focusing the past months with heavy use of rlhf. You can google official statements regarding "safety" of the model. I have a bunch of standard pre-prompt I have been using to initialize my chats since the beginning, and with time you could see how the model followed the instructions less strictly.
Problem with openai is that they never released exact number of parameters they are using and detailed benchmarks. And benchmarks you find online refer to APIs that behave differently than the chat webui (for instance you have longer context, you set temperature and system prompt, they are probably even different models, who knows... All is closed)
Measuring performances of llm is pretty tricky, minimal changes can have big effects (see https://huggingface.co/blog/evaluating-mmlu-leaderboard), and unfortunately I haven't found good resources to properly track chatgpt performances (from web ui) over time, across iterations
Thank you for the detailed reply 👍🏻
None of these points are true though. Context has been extended in the webui, markedly. 3.5 turbo is only that, 3.5 but faster. Gpt-4 is a marked improvement on 3.5 and I definitely haven’t seen any conclusive evidence it’s been nerfed in my daily use. Prompts have and still need to be carefully crafted for best results, but the results have been steadily improving not degrading over time.
All of these points are true though. Chatgpt 4 max token is now half of from the webui compared to when gtp-4 was launched. It used to be >8k, it is now >4k. Max number of tokens for the api hasn't changed for gpt-4, while it was greatly increased for chatgpt-3.5-turbo. The article is however talking about the service chatgpt, used via webui.
ChatGPT-3.5-turbo are different models than those used in the past. You can literally read it in the https://platform.openai.com/docs/models/gpt-3-5
Prompt engineering has been limited as demonstrated by the fact that most jailbreaking techniques don't work anymore. The way to avoid jailbreaking is exactly to limit ability of users to instruct the model.
Source on the halved token limit for gpt- 4 in the webui? Because that has not been my experience at all. There are now 16k and 32k models for 3.5-turbo, but there’s no evidence 3.5-turbo is nerfed at all from 3.5 and it absolutely out performs 3. Yes, you can see that they offer different snapshots of models, but that doesn’t indicate at all that there’s been a any reduction in their ability. “Breaking” jail breaking isn’t a bug, and it certainly hasn’t been demonstrated that the model is less capable.
Unless they reverted the chance recently (or using some regional A/B testing), you can test yourself the max number of tokens of gpt-4 from webui, that is now ~4k. It used to be ~ 8k.
What you are talking about are the APIs, that are different, and are not discussed in the news. They are even different models, in the sense that depending on the size of the context you get different results because of the attention mechanism. Unfortunately there is no official benchmark from openai as a comparison between gpt-3.5-turbo models with different context size, but I would not trust them much anyway. They are very defensive on their data, and push out mainly marketing stuff. I would wait for a 3rd party to do the benchmark.
"Breaking" jailbreaking is not a bug, but it limits the ability to instruct the model, i.e. prompt engineering, because it is literally meant to limit prompt engineering, it is the whole idea behind it
Edit. Here a link of a guide where they have the ~4k limit as well for gpt-4 https://the-decoder.com/chatgpt-guide-prompt-strategies/
I was skeptical at first but I've seen enough evidence now. There are definitely times when it's dumb as a brick, whether the filters just get in the way too much, or whether they've implemented other changes idk. I'd really love the unchained version.
On 23rd of March 2023 I asked a family member to give me a prompt and they asked "what day is 19th of April?".
It answered "The 19th of April falls on a Tuesday.", which was true last year but completely misleading if I thought we were taling about the coming month.
Was it wrong or just unclear? Either way it wasn't helpful.
I use it daily too and haven't had any of the issues I see written about it
I used the chatgpt site twice. Since then the Bing integration.
Is it rude to ask what you use it for?
Same. It saves me a lot of time both at work and when I'm working on my personal projects. But you need to ask proper questions to get proper answers.