Fail on gigantic (1m+ messages) logs

CyberFoxar commented

2024-07-10 10:17:40 +02:00

(Migrated from github.com)

Logs

I play on some RP server with a ton of fancy chat effects.

I'm unsure if that on your mod or something else, but I seem to be unable to re-open the enormous logs that this server generates. Upon trying to open one of those, I get:

-%<-
[09:34:58] [Render thread/ERROR]: Failed to load chat logs!
[09:34:58] [Render thread/INFO]: [STDERR]: java.io.EOFException: Unexpected end of ZLIB input stream
-%<-

Full log: https://mclo.gs/wmQQEJJ

I'll link a few of the problematic logs, and one working one (they're over 25Mb, so please excuse my external links, on my personal cloud, will expire in a month or so)

working log: https://cloud.foxwolf.casa/index.php/s/DRN3q6RtP8zrWrj
Causes the error: https://cloud.foxwolf.casa/index.php/s/mFazRLo8w5oHA73
Causes the error: https://cloud.foxwolf.casa/index.php/s/KZRzHz3D7f8ZSoi

Environement

As you might deduce from the above log, I'm running with Meteor Client 0.5.6 (albeit with the chat module deactivated), ChatPatches and Stendhal.

I am using version 0.2.3-build20240515 of Permanent-Chatlogs for minecraft 1.20.4

I'm also running Fabulously Optimized modpack, version 5.10.2, albeit with More Chat History disabled, as it conflicted with Meteor.

All 3 of the above logs have been made with the exact same mod setup.

Hypothesis / context / troubleshoots

I'm not sure if that error comes from the heavy log (>1Gb after uncompress) or from the various crashes I had during the session(s). This does not seems to be a memory problem, as I have plenty (15G) of memory allocated to minecraft, and plenty more available on my system.

VSCode has troubles opening the unzipped files, and most of my other text editors had similar issues. I only managed to open the file(s) when piping them in less, which let me confirm, albeit very slowly, that something was there, and that it looked like the expected JSON text.

Upon unzipping the log 7/8 with NanaZip, I get Unexpected end of data : log-8, which seems to indicate some sort of file corruption. However, the data itself seems to be (at least partially) there, I'm just not sure how to recover it, and would like some help on that front too.

I am noting that here because I believe this is an issue that you might be able to alleviate in future versions, maybe via splitting the logs more ? For now, I've reduced the save interval in the hope that it'll give me smaller files.

## Logs I play on some RP server with a ton of fancy chat effects. I'm unsure if that on your mod or something else, but I seem to be unable to re-open the enormous logs that this server generates. Upon trying to open one of those, I get: ```txt -%<- [09:34:58] [Render thread/ERROR]: Failed to load chat logs! [09:34:58] [Render thread/INFO]: [STDERR]: java.io.EOFException: Unexpected end of ZLIB input stream -%<- ``` Full log: https://mclo.gs/wmQQEJJ I'll link a few of the problematic logs, and one working one (they're over 25Mb, so please excuse my external links, on my personal cloud, will expire in a month or so) - working log: https://cloud.foxwolf.casa/index.php/s/DRN3q6RtP8zrWrj - Causes the error: https://cloud.foxwolf.casa/index.php/s/mFazRLo8w5oHA73 - Causes the error: https://cloud.foxwolf.casa/index.php/s/KZRzHz3D7f8ZSoi ## Environement As you might deduce from the above log, I'm running with Meteor Client 0.5.6 (albeit with the chat module deactivated), ChatPatches and Stendhal. I am using version `0.2.3-build20240515` of Permanent-Chatlogs for minecraft 1.20.4 I'm also running Fabulously Optimized modpack, version 5.10.2, albeit with `More Chat History` disabled, as it conflicted with Meteor. All 3 of the above logs have been made with the exact same mod setup. ## Hypothesis / context / troubleshoots I'm not sure if that error comes from the heavy log (>1Gb after uncompress) or from the various crashes I had during the session(s). This does _not_ seems to be a memory problem, as I have plenty (15G) of memory allocated to minecraft, and plenty more available on my system. VSCode has troubles opening the unzipped files, and most of my other text editors had similar issues. I only managed to open the file(s) when piping them in `less`, which let me confirm, albeit very slowly, that _something_ was there, and that it looked like the expected JSON text. Upon unzipping the log 7/8 with `NanaZip`, I get `Unexpected end of data : log-8`, which seems to indicate some sort of file corruption. However, the data itself seems to be (at least partially) there, I'm just not sure how to recover it, and would like some help on that front too. I am noting that here because I believe this is an issue that you might be able to alleviate in future versions, maybe via splitting the logs more ? For now, I've reduced the save interval in the hope that it'll give me smaller files.

👍 1

lovexyn0827 commented

2024-07-11 15:54:33 +02:00

(Migrated from github.com)

Thanks for your detailed report.

However, currently, I couldn't infer the exact cause of this bug from the provided information. It seems that the saved chatlogs are truncated at some point, probably as a consequence of the shortage of disk space, since autosaved chatlogs (they are only for recovering data after crashes) occupy approximately the same size as the unzipped data, which is about 4 GB here.

I'm not sure if that error comes from the heavy log (>1Gb after uncompress) or from the various crashes I had during the session(s).

Both are possible. I wonder if it is convenient to upload the log & crash reports generated when these problematic chatlogs are saved to confirm.

I'm just not sure how to recover it, and would like some help on that front too.

That is possible, actually simply suppressing exceptions to allow damaged chatlogs to be loaded should be helpful.

Also, I have to apologize for that this mod has quite a few problems when handling large chatlogs, as all messages are kept in the memory during entire sessions, even when autosaves are performed. Anyway, introducing better approaches to manage huge chatlogs should be prioritized.

Thanks for your detailed report. However, currently, I couldn't infer the exact cause of this bug from the provided information. It seems that the saved chatlogs are truncated at some point, probably as a consequence of the shortage of disk space, since autosaved chatlogs (they are only for recovering data after crashes) occupy approximately the same size as the unzipped data, which is about 4 GB here. > I'm not sure if that error comes from the heavy log (>1Gb after uncompress) or from the various crashes I had during the session(s). Both are possible. I wonder if it is convenient to upload the log & crash reports generated when these problematic chatlogs are saved to confirm. > I'm just not sure how to recover it, and would like some help on that front too. That is possible, actually simply suppressing exceptions to allow damaged chatlogs to be loaded should be helpful. Also, I have to apologize for that this mod has quite a few problems when handling large chatlogs, as all messages are kept in the memory during entire sessions, even when autosaves are performed. Anyway, introducing better approaches to manage huge chatlogs should be prioritized.

lovexyn0827 commented

2024-07-11 16:35:49 +02:00

(Migrated from github.com)

https://modrinth.com/mod/permanent-chatlogs/version/gApT6Qne

Hopefully this build can recover these damaged chatlogs.

[https://modrinth.com/mod/permanent-chatlogs/version/gApT6Qne](https://modrinth.com/mod/permanent-chatlogs/version/gApT6Qne) Hopefully this build can recover these damaged chatlogs.

CyberFoxar commented

2024-07-11 17:49:13 +02:00

(Migrated from github.com)

Many thanks, I'll try out your build tonight.

Edit: I can confirm that it loads the truncated logs just fine, though the scrollbar is a bit funky and the scroll behaves a bit erratically. It's very usable, and loads better than no logs x) Many thanks !

The lack of space might also have contributed indeed. Shortly after I submitted this issue, I realized I only had about 8Gig of space left on the disk that my instance lives on (which is easily filled up if multiple crashes occur closely (a quick look in my folder showed about 4 of those temporary 4Gig files). I will clear up some more space and keep a closer look on the disk usage. If I see the disk fill up, I'll look into moving my instances somewhere else.

Introducing better approaches to manage huge chatlogs should be prioritized.

Indeed ! I see a few potential "naive" fixes:

More flushing: Have the logs be archived more often, and flushed+zipped more often.
Manual flushing: Allow the user to execute a blocking save, flushing the currently saved logs to disk. I'd be fine with such a solution, it might even allow me to "separate" by conversation each log

I encourage you to add a few warnings about space and memory usage too.

I realize this is easy for me to say, and would like to reiterate: I'm glad for you mod and I'll try to provide as much help as I'm able ^^

Finally, and I realize this might be a bit out of place in a discussion about an issue (I'm willing to open another one, if needed) -- would you be able to build a log viewer that doesn't require launching minecraft ? Maybe an HTML page ?

If not, that's OK -- I'll probably take a shot at it, I really enjoy the thoroughness of your approach, and how perfectly it captures all the log !

Many thanks, I'll try out your build tonight. Edit: I can confirm that it loads the truncated logs just fine, though the scrollbar is a bit funky and the scroll behaves a bit erratically. It's very usable, and loads better than no logs x) Many thanks ! The lack of space might also have contributed indeed. Shortly after I submitted this issue, I realized I only had about 8Gig of space left on the disk that my instance lives on (which is easily filled up if multiple crashes occur closely (a quick look in my folder showed about 4 of those temporary 4Gig files). I will clear up some more space and keep a closer look on the disk usage. If I see the disk fill up, I'll look into moving my instances somewhere else. > Introducing better approaches to manage huge chatlogs should be prioritized. Indeed ! I see a few potential "naive" fixes: 1. More flushing: Have the logs be archived more often, and flushed+zipped more often. 2. Manual flushing: Allow the user to execute a blocking save, flushing the currently saved logs to disk. I'd be fine with such a solution, it might even allow me to "separate" by conversation each log I encourage you to add a few warnings about space and memory usage too. I realize this is easy for me to say, and would like to reiterate: I'm glad for you mod and I'll try to provide as much help as I'm able ^^ --- Finally, and I realize this might be a bit out of place in a discussion about an issue (I'm willing to open another one, if needed) -- would you be able to build a log viewer that doesn't require launching minecraft ? Maybe an HTML page ? If not, that's OK -- I'll probably take a shot at it, I really enjoy the thoroughness of your approach, and how perfectly it captures all the log !

lovexyn0827 commented

2024-07-12 09:47:13 +02:00

(Migrated from github.com)

Thanks for your suggestions!

Have the logs be archived more often, and flushed+zipped more often.

That would be OK. Although zipping large files while autosaving might be a little bit expensive, it is probably the best solution available. (maybe some settings should be introduced?)

Allow the user to execute a blocking save, flushing the currently saved logs to disk.

Good idea! But I wonder which method (like key bindings or separate buttons) is more suitable to start a manual save?

I'd be fine with such a solution, it might even allow me to "separate" by conversation each log.

Sounds like the solution in ReplayMod? I think it could be easily implemented by recreating new log files during sessions.

would you be able to build a log viewer that doesn't require launching minecraft ? Maybe an HTML page ?

This is exactly what I am going to do next. However, I don't think it is easy to build an external log viewer, without the help of Minecraft, since the whole text system should be rewritten to load texts. Maybe exporting chatlogs into some common file formats could be simpler.

I will be grateful for your help.

Thanks for your suggestions! > Have the logs be archived more often, and flushed+zipped more often. That would be OK. Although zipping large files while autosaving might be a little bit expensive, it is probably the best solution available. (maybe some settings should be introduced?) > Allow the user to execute a blocking save, flushing the currently saved logs to disk. Good idea! But I wonder which method (like key bindings or separate buttons) is more suitable to start a manual save? > I'd be fine with such a solution, it might even allow me to "separate" by conversation each log. Sounds like the solution in ReplayMod? I think it could be easily implemented by recreating new log files during sessions. > would you be able to build a log viewer that doesn't require launching minecraft ? Maybe an HTML page ? This is exactly what I am going to do next. However, I don't think it is easy to build an external log viewer, without the help of Minecraft, since the whole text system should be rewritten to load texts. Maybe exporting chatlogs into some common file formats could be simpler. I will be grateful for your help.

❤️ 1

CyberFoxar commented

2024-07-12 12:32:39 +02:00

(Migrated from github.com)

zipping large files while autosaving might be a little bit expensive

Yeah, that's my worry too. Unless there's a way to parallelize it, or do it in the background, it seems a bit more complex.

manual save

There's a few way that this could be implemented:

Button in the mod menu: This might be the easiest, but requires the user to click through a few screens before doing a save, which isn't ideal
Keybind: Probably the second-easiest method, and can be declined in a few way depending on how thorough we want that to be.
Button somewhere in one of the other screens
- given that this is a chat-related feature, maybe have a button somewhere on the screen when the chat window is opened ?
- This could also be in the pause menu, though that makes a lot less sense
client-side command: I am very unsure how feasable it might be, but I know that mods such as Meteor and Baritone can "register" client-side commands. This feels overkill for our usage, but I'm including it for thoroughness.

I'm leaning toward a keybind and/or a button on the chat ui.

As for the functionality, there are a few levels that we could think about, of increasing complexity:

Executing the action ends current log and saves it to disk
Executing the action pops up a menu where you can change a few attributes of the log (for example, giving it a name, and maybe a comment), kinda like how the "quick waypoint" menu from Xaero's minimap look.

"separate" by conversation each log.

I meant it in a more simple way. As conversation tend to happen one after another in a RolePlaying context, having either a marker or splitting the log (manually) when a conversation ends seems like a good idea, and would help reduce the risk of log corruption, at the cost of less efficient compression.

External viewer

The chatlogs are already in a common-ish format, which is JSON in minitext format, I believe ? The issue is more with how to parse them outside of minecraft, whilst still maintaining the looks and UX (buttons, hovers, etc). I have some experience with HTML/NodeJS and will probably try and play with that in the coming days -- Though I am probably starting a new job soon.

> zipping large files while autosaving might be a little bit expensive Yeah, that's my worry too. Unless there's a way to parallelize it, or do it in the background, it seems a bit more complex. > manual save There's a few way that this could be implemented: - Button in the mod menu: This might be the easiest, but requires the user to click through a few screens before doing a save, which isn't ideal - Keybind: Probably the second-easiest method, and can be declined in a few way depending on how thorough we want that to be. - Button somewhere in one of the other screens - given that this is a chat-related feature, maybe have a button somewhere on the screen when the chat window is opened ? - This could also be in the pause menu, though that makes a lot less sense - client-side command: I am very unsure how feasable it might be, but I know that mods such as Meteor and Baritone can "register" client-side commands. This feels overkill for our usage, but I'm including it for thoroughness. I'm leaning toward a keybind and/or a button on the chat ui. As for the functionality, there are a few levels that we could think about, of increasing complexity: 1. Executing the action ends current log and saves it to disk 2. Executing the action pops up a menu where you can change a few attributes of the log (for example, giving it a name, and maybe a comment), kinda like how the "quick waypoint" menu from Xaero's minimap look. > "separate" by conversation each log. I meant it in a more simple way. As conversation tend to happen one after another in a RolePlaying context, having either a marker or splitting the log (manually) when a conversation ends seems like a good idea, and would help reduce the risk of log corruption, at the cost of less efficient compression. > External viewer The chatlogs are _already_ in a common-ish format, which is JSON in minitext format, I believe ? The issue is more with how to parse them outside of minecraft, whilst still maintaining the looks and UX (buttons, hovers, etc). I have some experience with HTML/NodeJS and will probably try and play with that in the coming days -- Though I am probably starting a new job soon.

lovexyn0827 commented

2024-07-13 02:34:44 +02:00

(Migrated from github.com)

Unless there's a way to parallelize it

Autosaving has already been put into separate threads in current versions. But the main worry is whether or not zipping a large file would use up the memory, which should be tested first.

I'm leaning toward a keybind and/or a button on the chat ui.

Both ways are okay, but keybindings are much easier to implement.

Executing the action ends current log and saves it to disk

That will work. Probably I will add the ability to pause recording and to save without terimating a session by the way.

having either a marker or splitting the log (manually) when a conversation ends seems like a good idea

This is exactly what I am going to work on in the future!

The chatlogs are already in a common-ish format, which is JSON in minitext format

Yes, but exporting chatlogs as TXT or HTML should be also useful, I think. I will work on this when I get some spare time.

I have some experience with HTML/NodeJS and will probably try and play with that in the coming days

Many thanks. I am so glad for your help.

> Unless there's a way to parallelize it Autosaving has already been put into separate threads in current versions. But the main worry is whether or not zipping a large file would use up the memory, which should be tested first. > I'm leaning toward a keybind and/or a button on the chat ui. Both ways are okay, but keybindings are much easier to implement. > Executing the action ends current log and saves it to disk That will work. Probably I will add the ability to pause recording and to save without terimating a session by the way. > having either a marker or splitting the log (manually) when a conversation ends seems like a good idea This is exactly what I am going to work on in the future! > The chatlogs are already in a common-ish format, which is JSON in minitext format Yes, but exporting chatlogs as TXT or HTML should be also useful, I think. I will work on this when I get some spare time. > I have some experience with HTML/NodeJS and will probably try and play with that in the coming days Many thanks. I am so glad for your help.

❤️ 1

lovexyn0827 commented

2024-08-31 09:06:13 +02:00

(Migrated from github.com)

Finally, the code to save chat logs has been completely rewritten. Chat logs are no longer cached in the memory, and autosaving is no longer needed since the chat logs are written at real time now.

Hopefully this will fix the issue. You may test with this version .

Finally, the code to save chat logs has been completely rewritten. Chat logs are no longer cached in the memory, and autosaving is no longer needed since the chat logs are written at real time now. Hopefully this will fix the issue. You may test with [this version](https://modrinth.com/mod/permanent-chatlogs/version/ZPHCxm75) .

CyberFoxar commented

2024-09-01 20:39:24 +02:00

(Migrated from github.com)

Alright ! I'm running this version now. We'll see how it goes.

lovexyn0827 commented

2024-10-12 16:59:39 +02:00

(Migrated from github.com)

New version here~

https://modrinth.com/mod/permanent-chatlogs/version/0.3.0-build20241012

New version here~ https://modrinth.com/mod/permanent-chatlogs/version/0.3.0-build20241012

👍 1