Magnetometer hard iron / soft iron calibration-compensation

Question

I have real magnetometer data captured by BMX055. I deployed an hard-soft iron calibration code (magcal) on this data in Matlab. Here is the code:

addpath('quaternion_library'); clear all;close all;clc;load('ExampleData.mat');

%Ideal Magnetometer Data----------------------------------------------------
 
N = 10000;rng(1);acc = zeros(N,3);av = zeros(N,3);
q = randrot(N,1); % uniformly distributed random rotations
imu = imuSensor('accel-mag');[~,x] = imu(acc,av,q);

scatter3(x(:,1),x(:,2),x(:,3));axis equal;title('Ideal Magnetometer Data');
 
%Magnetometer Data With a Hard Iron Offset----------------------------------
 
imu.Magnetometer.ConstantBias = [2 10 40];[~,y] = imu(acc,av,q);
figure;scatter3(y(:,1),y(:,2),y(:,3));axis equal;title('Magnetometer Data With a Hard Iron Offset');
 
%Magnetometer Data With Hard and Soft Iron Effects--------------------------
 
nedmf = imu.MagneticField;Rsoft = [2.5 0.3 0.5; 0.3 2 .2; 0.5 0.2 3];
soft = rotateframe(conj(q),rotateframe(q,nedmf)*Rsoft);
 
for ii=1:numel(q)
    imu.MagneticField = soft(ii,:);
    [~,z(ii,:)] = imu(acc(ii,:),av(ii,:),q(ii));
end
 
figure;scatter3(z(:,1),z(:,2),z(:,3));axis equal;title('Magnetometer Data With Hard and Soft Iron Effects');
 
%+ Noise ---------------------------------------------------------------------
 
imu.Magnetometer.NoiseDensity = 0.08;
for ii=1:numel(q)
    imu.MagneticField = soft(ii,:);
    [~,t(ii,:)] = imu(acc(ii,:),av(ii,:),q(ii));
end
 
%Magnetometer Calibrated (Hard and Soft Iron Effects Cal)----------------------
 
[A,b,expMFS]  = magcal(t);xCorrected = (t-b)*A;%r=z-xCorrected;

%Magnetometer Calibrated Plot---------------------------------------------------
 
figure;scatter3(xCorrected(:,1),xCorrected(:,2),xCorrected(:,3));axis equal;title('Magnetometer Data Calibrated');
 
%Magnetometer Calibration Residual error
r = sum(xCorrected.^2,2) - expMFS.^2;
E = sqrt(r.'*r./N)./(2*expMFS.^2);
fprintf('Residual error in corrected data : %.2f\n\n',E);

Although Matlab generic data is well calibrated as shown here: bias-offset-cross axis misalignment of my real data (Magno1.csv) has not been calibrated by the same code:

Why so? I think i got sufficient data, so magcal function should fit its ellipsoid to sphere like in the matlab data. what could be the reason?

Thanks..

p.s.1: Magno1.csv: https://docs.google.com/spreadsheets/d/1kZVBB7e2zEP52UwmvabkfaAnA7M6ttjVnNGp7hFQcr4/edit#gid=1676163070

p.s.2: For [A,b,expMFS] = magcal(t), 3x3 A matrices are same for both, only b matrices have different offset values.