Copied bc-fips-1.0.2.jar to {JRE_HOME}/lib/ext folder.

Updated the {JRE_HOME}/lib/security/java.security property file with the below details-

 security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
 security.provider.2=com.sun.net.ssl.internal.ssl.Provider BCFIPS
 security.provider.3=sun.security.provider.Sun
 security.provider.4= ...
 # Default keystore type.
 keystore.type=BCFKS
Updated {CATALINA_HOME}/bin/setenv.bat with the below details-
SET SSL_OPTS=%SSL_OPTS% -Djavax.net.ssl.trustStore=D:\..\apache-tomcat-9.0.19\data\properties/tomcatTrustStore_BCFKS.bcfks
SET SSL_OPTS=%SSL_OPTS% -Djavax.net.ssl.trustStorePassword=changeit
SET SSL_OPTS=%SSL_OPTS% -Djavax.net.ssl.keyStore=D:\..\apache-tomcat-9.0.19\data\security\jks/h2_keystore_BCFKS.bcfks
SET SSL_OPTS=%SSL_OPTS% -Djavax.net.ssl.keyStorePassword=changeit
SET JAVA_OPTS=%JAVA_OPTS% -Djava.security.debug=all -Dorg.bouncycastle.fips.approved_only=true -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Note- I have imported my existing trust store and key store in jks format to bcfks format using the below commands-
keytool -importkeystore -srckeystore D:\myapp\app\properties\tomcatTrustStore.jks -srcstoretype JKS -deststoretype BCFKS -destkeystore D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\data\properties\tomcatTrustStore_BCFKS.bcfks -srcstorepass changeit -deststorepass changeit -providerpath "D:\myapp\jdk1.8.0_192\jre\lib\ext\bc-fips-1.0.2.jar" -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
keytool -importkeystore -srckeystore D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\data\security\jks\h2_keystore.jks -srcstoretype JKS -deststoretype BCFKS -destkeystore D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\data\security\jks\h2_keystore_BCFKS.bcfks -srcstorepass changeit -destkeypass changeit -deststorepass changeit -providerpath "D:\myapp\jdk1.8.0_192\jre\lib\ext\bc-fips-1.0.2.jar" -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
I have a listener class that starts up h2 db server when we start our tomcat server. However, the h2 db server is failing to start with the error-
Caused by: java.io.IOException: DER length more than 4 bytes: 109
    at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
    at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
    at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source)
    at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source)
    at java.security.KeyStore.load(KeyStore.java:1445)
The full stack trace is-
2021-06-18 02:24:47,309 INFO org.apache.catalina.core.StandardService - Starting service [tomcatapp]
2021-06-18 02:24:47,314 INFO org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.19]
2021-06-18 02:24:47,445 INFO org.apache.catalina.startup.HostConfig - Deploying deployment descriptor [D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\conf\CatalinaSSL\localhost\tomcatapp.xml]
2021-06-18 02:24:47,516 WARNING org.apache.catalina.startup.HostConfig - Deployment of deployment descriptor [D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\conf\CatalinaSSL\localhost\tomcatapp.xml] with an external docBase means the directory [D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19\stpapps\tomcatapp] in the appBase will be ignored
2021-06-18 02:24:47,620 INFO com.company.tomcatapp.listener.H2LifecycleListener - Initializing H2 tcp server on Tomcat lifecycle: STARTING_PREP
2021-06-18 02:24:47,662 INFO com.company.tomcatapp.service.impl.H2TcpServerServiceImpl - Created H2 server with the following parameters: [-tcpPort, 9094, -tcpAllowOthers, -baseDir, D:\myapp\tomcatapp-0.0.249\apache-tomcat-9.0.19/data/db]
2021-06-18 02:24:47,672 INFO com.company.tomcatapp.service.impl.H2TcpServerServiceImpl - Starting H2 server...
2021-06-18 02:24:48,416 INFO com.company.tomcatapp.service.impl.H2TcpServerServiceImpl - H2 server started
2021-06-18 02:24:48,466 INFO com.company.tomcatapp.listener.H2LifecycleListener - Initializing H2 web server on Tomcat lifecycle: STARTING_PREP
2021-06-18 02:24:48,595 INFO com.company.tomcatapp.service.impl.H2WebServerServiceImpl - Created H2 server with the following parameters: [-webSSL, -webPort, 8084, -webAllowOthers]
2021-06-18 02:24:48,601 INFO com.company.tomcatapp.service.impl.H2WebServerServiceImpl - Starting H2 server...
2021-06-18 02:24:49,108 SEVERE com.company.tomcatapp.service.impl.H2WebServerServiceImpl - Failed to start H2 server!
org.h2.jdbc.JdbcSQLException: IO Exception: "java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)"; "port: 8084 ssl: true" [90031-176]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
    at org.h2.message.DbException.get(DbException.java:167)
    at org.h2.message.DbException.convertIOException(DbException.java:329)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:198)
    at org.h2.util.NetUtils.createServerSocket(NetUtils.java:161)
    at org.h2.server.web.WebServer.start(WebServer.java:354)
    at org.h2.tools.Server.start(Server.java:476)
    at com.company.tomcatapp.service.AbstractH2ServerService.startServer(AbstractH2ServerService.java:45)
    at com.company.tomcatapp.listener.H2LifecycleListener.lifecycleEvent(H2LifecycleListener.java:53)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:631)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1831)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:425)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
    at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:929)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902)
    at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
    at javax.net.ssl.DefaultSSLServerSocketFactory.throwException(SSLServerSocketFactory.java:160)
    at javax.net.ssl.DefaultSSLServerSocketFactory.createServerSocket(SSLServerSocketFactory.java:173)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:121)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:188)
    ... 46 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
    at java.security.Provider$Service.newInstance(Provider.java:1617)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
    at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
    at javax.net.ssl.SSLServerSocketFactory.getDefault(SSLServerSocketFactory.java:113)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:118)
    ... 47 more
  Caused by: java.io.IOException: DER length more than 4 bytes: 109
    at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
    at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
    at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source)
    at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
    at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:877)
    at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:854)
    at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:1019)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.security.Provider$Service.newInstance(Provider.java:1595)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
    at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
    at javax.net.ssl.SSLServerSocketFactory.getDefault(SSLServerSocketFactory.java:113)
    at org.h2.security.CipherFactory.createServerSocket(CipherFactory.java:118)
    at org.h2.util.NetUtils.createServerSocketTry(NetUtils.java:188)
    at org.h2.util.NetUtils.createServerSocket(NetUtils.java:158)
    ... 45 more
Could someone please guide me on what could be the possible reason for this issue?
                DER means Distinguished Encoding Rules. This is the encoding of your ASN1 files. But because your question does not include any information about your DER files files, we can not answer it. DER files are typically certificates or private keys unless encoded in PEM.
– ceving
                Jun 18, 2021 at 13:44
                @ceving I have added details about the keystore files and the command using which I have converted it from jks to bcfks format. Please let me know what else information you need.
– Harsh Raj
                Jun 18, 2021 at 15:15
                Both your -importkeystore commands create files in .../data/security/jks/ but your catalina setenv uses one (apparently) from there and one from ../data/properties -- are you sure (both) the files your server is reading are the files you created?
– dave_thompson_085
                Jun 18, 2021 at 21:30
                @dave_thompson_085 Actually I am copying it later to the ../data/properties folder. Let me correct it above in the question. Thanks for looking into it. Any further help would be appreciated.
– Harsh Raj
                Jun 19, 2021 at 8:00
                Sorry, if it's indeed the correct file I can't help. I don't and won't use bcfips so I don't know what the structure should be, but if I did and had this error I would look a hexdump, compare to an (attempted) ASN.1 parse using OpenSSL, and compare that to the source.
– dave_thompson_085
                Jun 19, 2021 at 10:41
You can try adding this in your setenv.sh file, this worked for me.
export JAVA_OPTS="Djavax.net.ssl.trustStore=truststore_path/truststore.bcfks -Djavax.net.ssl.trustStorePassword=truststorepassword -Djavax.net.ssl.trustStoreType=BCFKS -Djavax.net.ssl.keyStore=keystore_path/keystore.bcfks -Djavax.net.ssl.keyStorePassword=keystorepassword -Djavax.net.ssl.keyStoreType=BCFKS"
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.