How to repair corrupted integrity of a JetBrains Xodus database?

Question

Background: Our YouTrack database got corrupted. The relevant bug report is here: https://youtrack.jetbrains.com/issue/JT-48235#comment=27-2952375 Now I'm trying to fix the corrupted state of the Xodus database that YouTrack uses internally.

Problem: The database contains entities that have links to deleted entities. My question is: How do I delete or repair these broken links?

After shutting down YouTrack I used Xodus Entity Browser to inspect the database.

I came up with two possible solutions:

Recreate the deleted Entity. In my case that was Sprint[86-191]. This did not work because creating a new Entity did not allow me to choose the number 191 again. Instead it created Sprint[86-208].
Remove the broken link. Did not work because in order to remove the link, Xodus expects consistency of the database resulting in the following stack trace:

jetbrains.xodus.browser.web.EntityNotFoundException: jetbrains.exodus.entitystore.EntityRemovedInDatabaseException: Sprint was removed.                                                       
        at jetbrains.xodus.browser.web.db.StoreService.getEntity(StoreService.kt:204) ~[xodus-entity-browser.jar:na]                                                                          
        at jetbrains.xodus.browser.web.db.StoreService.access$getEntity(StoreService.kt:14) ~[xodus-entity-browser.jar:na]                                                                    
        at jetbrains.xodus.browser.web.db.StoreService$getEntity$1.invoke(StoreService.kt:168) ~[xodus-entity-browser.jar:na]                                                                 
        at jetbrains.xodus.browser.web.db.StoreService$getEntity$1.invoke(StoreService.kt:14) ~[xodus-entity-browser.jar:na]                                                                  
        at jetbrains.xodus.browser.web.db.StoreServiceKt$transactional$1.compute(StoreService.kt:229) ~[xodus-entity-browser.jar:na]                                                          
        at jetbrains.exodus.entitystore.PersistentEntityStoreImpl.computeInTransaction(PersistentEntityStoreImpl.java:596) ~[xodus-entity-browser.jar:na]                                     
        at jetbrains.xodus.browser.web.db.StoreServiceKt.transactional(StoreService.kt:229) ~[xodus-entity-browser.jar:na]                                                                    
        at jetbrains.xodus.browser.web.db.StoreService.transactional(StoreService.kt:218) ~[xodus-entity-browser.jar:na]                                                                      
        at jetbrains.xodus.browser.web.db.StoreService.getEntity(StoreService.kt:167) ~[xodus-entity-browser.jar:na]                                                                          
        at jetbrains.xodus.browser.web.resources.Entities$registerRouting$1$2.invoke(Entities.kt:37) ~[xodus-entity-browser.jar:na]                                                           
        at jetbrains.xodus.browser.web.resources.Entities$registerRouting$1$2.invoke(Entities.kt:10) ~[xodus-entity-browser.jar:na]                                                           
        at jetbrains.xodus.browser.web.WebConfigKt$safeGet$1.invoke(WebConfig.kt:61) ~[xodus-entity-browser.jar:na]                                                                           
        at jetbrains.xodus.browser.web.WebConfigKt$safeGet$1.invoke(WebConfig.kt) ~[xodus-entity-browser.jar:na]                                                                              
        at spark.kotlin.Http$get$1.handle(Http.kt:560) ~[xodus-entity-browser.jar:na]
        at spark.RouteImpl$1.handle(RouteImpl.java:72) ~[xodus-entity-browser.jar:na]
        at spark.http.matching.Routes.execute(Routes.java:61) ~[xodus-entity-browser.jar:na]
        at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130) ~[xodus-entity-browser.jar:na]                                                                                  
        at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50) [xodus-entity-browser.jar:na]                                                                               
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1568) [xodus-entity-browser.jar:na]                                                                    
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [xodus-entity-browser.jar:na]                                                                        
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [xodus-entity-browser.jar:na]                                                                      
        at org.eclipse.jetty.server.Server.handle(Server.java:564) [xodus-entity-browser.jar:na]                                                                                              
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) [xodus-entity-browser.jar:na]                                                                                    
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [xodus-entity-browser.jar:na]                                                                          
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [xodus-entity-browser.jar:na]                                                          
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [xodus-entity-browser.jar:na]                                                                                    
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [xodus-entity-browser.jar:na]                                                                                 
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) [xodus-entity-browser.jar:na]                                                                          
        at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) [xodus-entity-browser.jar:na]                                                                 
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) [xodus-entity-browser.jar:na]                                                             
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199) [xodus-entity-browser.jar:na]                                                                   
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673) [xodus-entity-browser.jar:na]                                                                     
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591) [xodus-entity-browser.jar:na]                                                                      
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: jetbrains.exodus.entitystore.EntityRemovedInDatabaseException: Sprint was removed.
        at jetbrains.exodus.entitystore.PersistentStoreTransaction.getEntity(PersistentStoreTransaction.java:276) ~[xodus-entity-browser.jar:na]                                              
        at jetbrains.xodus.browser.web.db.StoreService.getEntity(StoreService.kt:201) ~[xodus-entity-browser.jar:na]                                                                          
        ... 33 common frames omitted

Is there a way to solve this corrupted state?

BTW, it makes sense run YouTrack with additional java parameter `-Dexodus.entityStore.debug.testLinkedEntities=true`. Your case seems to be covered by this parameter: with such application errors, the database code will fail-fast instead of creating inconsistencies. We're now considering the possibility to force it for all YouTrack standalone instances. — Vyacheslav Lukianov, Jul 13 '18 at 13:20

score 3 · Accepted Answer · answered Jul 13 '18 at 12:34

3

Start YouTrack with the java parameter -Dexodus.entityStore.refactoring.heavyLinks=true. You don't need to force this refactoring on each start.

answered Jul 13 '18 at 12:34

Vyacheslav Lukianov

1,913
8
12

This indeed resolved the problem. Adding the java parameter to youtrack didn't work however. Starting the Xodus Entity Browser with that parameter and removing the default `-Dexodus.entityStore.refactoring.skipAll=true` did the trick. Thank you so much for the quick support! – Dons Jul 13 '18 at 13:06

How to repair corrupted integrity of a JetBrains Xodus database?

1 Answers1